Actively Searching for an Effective Neural Network Ensemble

Actively Searching for an E ective Neural-Network Ensemble David W. Opitz Jude W. Shavlik Computer Science Department Computer Sciences Department University of Minnesota University of Wisconsin 10 University Drive 1210 W. Dayton St. Duluth, MN 55812 Madison, WI 53706 [email protected] [email protected] 218-726-6149 608-262-7784 Fax: 218-726-8240 Abstract A neural-network ensemble is a very successful technique where the outputs of a set of separately trained neural network are combined to form one uni ed prediction. An e ective ensemble should consist of a set of networks that are not only highly correct, but ones that make their errors on di erent parts of the input space as well; however, most existing techniques only indirectly address the problem of creating such a set. We present an algorithm called Addemup that uses genetic algorithms to explicitly search for a highly diverse set of accurate trained networks. Addemup works by rst creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are highly accurate while disagreeing with each other as much as possible. Experiments on four real-world domains show that Addemup is able to generate a set of trained networks that is more accurate than several existing ensemble approaches. Experiments also show that Addemup is able to e ectively incorporate prior knowledge, if available, to improve the quality of its ensemble.